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Abstract. Even though experimental 
design (ED) in open inquiry is important 
for more authentic scientific inquiry, it is 

not easy for students. The objective of this 
research was to identify factors considered 
important in students’ ED and explore 

how to utilize it to improve students’ ED. 

To achieve the objective, in the first step, 
eighteen middle-school students in a gifted 
center participated in this study voluntar- 
ily. They were asked to design experimental 
procedures without any guidelines or help. 
Students’ EDs were analyzed qualitatively 
to extract difficulties that students experi- 
enced when designing experimental proce- 
dures. Based on analysis results, 17 factors 
were identified as factors to be considered 
for students’ ED and structured in a form of 
checklist. In the second step, the checklist 
was provided to 50 students in the gifted 
center as guidelines for their ED. They were 
then asked to design experimental proce- 
dures. Comparing scores of these students’ 
ED with those of the control group, it was 
found that these factors were significantly 
effective for improving students’ ED with 
high effect size. In conclusion, results of this 
research suggest that effective teaching 
strategy for students’ ED in scientific open 
inquiry can be obtained based on analysis 
about their difficulties in ED. 
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Introduction 


The aim of this research is to identify difficulties that students experience 
when they try to design experimental procedures in scientific open inquiry 
and obtain teaching strategy to improve students’ experimental design (ED) 
by considering these difficulties. 

Science educators have emphasized student-centered or open-type 
scientific inquiry in which students will find an inquiry problem, design 
experimental procedures, and draw conclusions by themselves (Kim & Park, 
2015; Berg, Bergendahl, Lundberg, & Tibell, 2003; Krystyniak & Heikkinen, 
2007; Zion, 2008; Zion & Mendelovici, 2013). Such open inquiry is important 
not only for elementary and high-school students, but also important for 
science teachers who guide these students (Lamanauskas & Augiene, 2016; 
Tatar, 2012; Windschitl, 2004). 

ED as one of major elements in open inquiry has also been emphasized 
in science curriculum. For example, National Science Education Standards 
in the USA have included “planning and carrying out investigations” as one 
of eight practices (inquiry skills) for K-12 science (National Research Council 
[NRC], 2012, p. 42). In Korea, the new science curriculum revised in 2015 em- 
phasizes that “Science courses should enable diverse inquiry-based learning”. 
It also included “designing and conducting inquiry” as one major inquiry skill 
(Ministry of Education [MOE], 2015, pp. 4-5). Actual scientists also emphasize 
the importance of ED. For example, Coil, Wenderoth, Cunningham, and Dirks 
(2010) have reported that ED is ranked the 7th or 8th among 22 scientific 
inquiry skills considered important in scientific research by 159 biologists. 

However, designing experimental procedures is not easy for students 
(Hugerat, Najami, Abbasi, & Dkeidek, 2014; Kim & Song, 2012; Lim, Yang, Kim, 
Hong, & Lim, 2010; Trautmann, McKinster, & Avery, 2004). In a preliminary 
stage of this research, 171 middle-school students enrolled in scientifically 
gifted centers in Korea were asked what parts were the most difficult ones 


Journal of Baltic Science Education, Vol. 16, No. 6, 2017 


ISSN 1648-3898 /Print/ IDENTIFYING AND APPLYING FACTORS CONSIDERED IMPORTANT IN STUDENTS’ 
EXPERIMENTAL DESIGN IN SCIENTIFIC OPEN INQUIRY 
ISSN 2538-7 138 /onines (P. 932-945) 


or needing help the most when they were conducting open scientific inquiries. As a result, 35% of them answered 
that ED was the most difficult part in open inquiry while 40% of them required help or guidance in designing ex- 
perimental procedures. Scientifically gifted students are more than average in science. They particularly have high 
interest in science. They also have many experiences in scientific inquiry. Nonetheless, their difficulties in designing 
experimental procedures indicate that extra special guidance is needed for ED. 

To improve students’ ability to design experimental procedures, many science educators have made various 
efforts. For example, Ross and Robinson (1987) have helped ninth- and tenth- grade students’ ED by using five 
guidelines to identify dependent variables, independent variables, controlling variables, and so on. Hiebert (2007) 
has conducted a study to help university students to design biological experiments, including controlling variables, 
determining sample size, and analyzing data statistically. Girault and Durham (2014) have used computers to im- 
prove students’ EDs by identifying errors in their design processes and providing feedbacks on them. 

Others have concerns about the development and application of tools for evaluating students’ ability to 
design experiments. Deane, Nomme, Jeffery, Pollock, and Birol (2014) have developed Biological Experimental 
Design Concept Inventory (BEDCI) consisting of 14 items in 8 categories to assess such abilities in biology. Sirum 
and Humburg (2011) have developed Experimental Design Ability Test (EDAT) consisting of 10 items. Etkina et al. 
(2006) have included “ability to design experimental inquiry” in a tool developed to assess scientific ability in the 
area of physics. This tool is relatively simple. It is consisted of only a few characteristic factors such as identifying 
dependent and independent variables, selecting appropriate experimental equipment, and clarifying disadvan- 
tages of ED. Kim and Kang (2014) have utilized Diet COLa Test (DCT) developed by Fowler (1990) to compare ED 
abilities of ordinary Korean students and students classed as gifted in science. 

This research was designed by combining the test of with the improvement of students’ EDs in science. To 
achieve this objective, students’ incomplete or inappropriate parts when designing experimental procedures were 
identified first. Attempts were then made by developing a checklist to improve students’ EDs by guiding them to 
revise their original designs. Finally, the effectiveness of the checklist for student’ ED was then tested. 

To help students overcome difficulties in EDs, there are different approaches. Ross and Robinson (1987) have 
used three approaches: (1) observe students’ EDs and help them when they face difficulties, (2) ask students to 
reflect on their own activities in designing experiments by using the checklist and indicate what to consider in ED, 
(3) ask students to find out what should be included in ED examples and use their findings in their ED. As a result, 
Ross and Robinson (1987) have concluded that these three approaches are equally effective. 

The first approach requires teacher to observe ED for each student and help each student individually. It is not 
easy to apply it to many students in an ordinary school laboratory. The third approach requires too much active 
participation by students because they have to find their own solutions for their ED problems. Regarding the third 
approach, Kirschner, Sweller, and Clark (2006) have reported that having students construct their own solutions 
without guided instructions is not so effective. Therefore, the second approach was used in the present research. 

That is, in this research, a checklist developed to improve students’ ED was provided to students, and then 
students were asked to use the checklist reflectively when carrying out their actual EDs. 

In the process of developing the checklist, two approaches (explicit and implicit) need to be considered. In the 
explicit approach, when a student conducts any scientific task, a direct guidance or specific teaching is provided for 
the given task. In the explicit approach, students learn specific skills or concepts in a specific situation or context 
of a given task. In this case, when the situation or context of a task changes, contents or skills to be learned also 
need to change. Through this process, students are ‘expected’ to improve their general skills or understanding 
independent of the context. The explicit approach can be viewed as an inductive process. 

On the other hand, the implicit approach can be viewed as a deductive process. In the implicit approach, 
teachers or teaching materials are focused on developing basic and general skills and abilities not limited to the 
context of the task, with the expectation that improved skills and abilities will be utilized in conducting the task 
in different contexts. 

Even though each approach has its own advantages, it has been reported that the implicit approach has a 
limitation in that the “expectation” might not work successfully. For example, in efforts to help students understand 
the nature of science (NOS), many science educators recommend an explicit approach because implicit teaching is 
relatively ineffective in improving NOS understanding (Abd-El-Khalick & Lederman, 2000; Khishfe & Abd-El-Khalick, 
2002; Schwartz, Lederman, & Crawford, 2004). Therefore, the present research will employ the explicit approach for 
students’ EDs. Instead of having general drills or exercises in arbitrary situations, direct guidance will be provided 
when students are trying to design experimental procedures in actual situations. 
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With the above background, this research is designed to identify students’ difficulties when they design ex- 
perimental procedures by extracting incomplete or unsuitable parts from their EDs. Based on these difficulties, a 
checklist will be developed to guide students’ actual ED explicitly. Its effectiveness is then tested. Detailed research 
goals are shown below: 

1) To identify students’ difficulties in EDs, 
2) To develop a checklist as a guideline for students’ EDs by considering students’ difficulties, 
3) To evaluate the effectiveness of the checklist by applying it to students’ actual EDs. 


Preliminary Research 
Methodology 


The purpose of preliminary research is to examine what difficulties students will have and what parts they 
need help in scientific open inquiry. A survey of was conducted for 171 students and their responses were analyzed. 


Sample 


At G city in Korea, there is a science gifted center under ’G city Education Office’ The center selects students 
who are gifted in science and mathematics from elementary and junior-high schools. It provides more than 100 
hours of study per year using weekends and vacations. Educational programs at the center are developed by school 
teachers or educational professionals separately from the national science curriculum. In general, there are about 
180 students registered at the center. Of these, a total of 171 students (76 seventh graders, 59 eighth graders, and 
36 ninth graders) voluntarily participated in this research after excluding 9 students who could not participate 
due to personal reasons. 


Questionnaire 


The following two questions were answered freely by the students: (1) On what part do you have difficulty in 
conducting open scientific inquiry? (2) What part do you need help with when conducting open scientific inquiry? 


Data analysis 

Students’ written responses were divided into the following six categories: 1) finding an inquiry problem, 2) 
designing experimental procedure, 3) conducting experiment (e.g., setting experimental equipment, obtaining 
data, and analyzing data), 4) allotting roles (including cooperation), 5) writing report, and 6) presenting inquiry 
results. The frequency of each category was then obtained. 

Results 

The part that students feel difficult in open inquiry 

The part that students feel difficult when conducting open inquiry is shown in Table 1. Many students answered 


that finding the inquiry problem (58%) and designing experimental procedure (35%) were the most difficult parts 
when conducting an open inquiry. 
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Finding 
inquiry 
problem 
f % 
36 47 
My 70 
22 61 
99 58 


Designing 
experimental 

procedure 

f % 
28 37 
16 27 
16 44 
60 35 
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27 


Conducting 
experiment 


The part that students feel difficult in open inquiry. 


Allocating roles 


% f 
18 0 
19 3 
6 2 
16 5 


Part of open inquiry 


% 


Note: Sum of responses exceeded 100% because some students gave multiple responses. 


The part that students need help with in an open inquiry 


Writing report 
f % 
0 0 
2 3 
0 0 
2 1 


Presenting 

the results 
f % 
10 13 
9 15 
2 6 
21 12 


Table 2 shows that part that students need help with in an open inquiry. Many students responded that they 
needed help for designing an experimental procedure (40%), finding an inquiry problem (30%), and presenting 


inquiry results (22%). 


Table 2. 
Grade 
f 
7(n=76) 16 
8(n=59) 20 
Q(n=36) 15 
Sum 5 


(n= 171) 


Note: Sum of responses exceeded 100% because some students gave multiple responses. 


Finding 
inquiry 
problem 


% 


21 
34 
42 


30 


Designing 
experimental 
procedure 
f % 
34 45 
23 39 
12 33 
69 40 


Conducting 

experiment 
f % 
10 13 
10 17 
7 19 
27 16 


The part that students need help with in open inquiry. 


Allocating roles 


Part of open inquiry 


5 


14 


% 


14 


Writing report 
f % 
0 0 
2 3 
1 3 
3 2 


Presenting 
the results 
f % 
12 16 
16 27 
9 25 
37 22 


Results of the preliminary research revealed that the most difficult and necessary parts for students in open 
inquiry were ‘problem finding’ and ‘experimental design’ For problem finding, Park (2005) has identified several 
strategies that can be effectively used to find inquiry problems. Park (2013) has conducted a research to improve 
students’ ability of finding inquiry problems based on results of Park (2005) and obtained positive results. Therefore, 


the present research focused on students’ EDs. 


Methodology of the Main Research 


Based on results of the preliminary research, the main research was focused on identifying students’ difficulties 
in EDs in more detail, and improving their ability of ED. The main research consisted of two stages. In the first stage, 
survey was conducted to find out more details of students’ difficulties when they needed to design experimental 
procedures. A checklist was then developed to guide students’ EDs based on findings about students’ difficulties. 
In the second stage of the main research, the checklist was applied to improve students’ ED ability and the effec- 


tiveness of it was evaluated. 
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Methodology the First Stage 
Sample 


Of students participating in the preliminary research, eighteen ninth graders who were taught by one of the 
authors of this research at the center were randomly selected. They voluntarily participated in this research. Con- 
ducing open inquiry where students set their own inquiry problems, design experiments, and perform inquiries 
is a major part of the gifted program in the center. Therefore, all students who participated in this research were 
willing to participate in ED activities. 


Task 


Six tasks were proposed to students for designing experimental procedures. Students chose one task freely 
and wrote up their experimental procedures on paper for 45 minutes without any guidance or help. The following 
tasks were the one that they could choose from: 

e Exploring factors affecting flight distance of a paper plane. 
e Exploring factors that make a top spin longer. 

e Reducing external noise at home. 

e¢ Measuring elastic coefficient of different materials. 

e Designing a robust (strong) bridge. 

e Designing earthquake-resistant buildings. 


Development of a checklist 

From students’ EDs, incomplete and inappropriate parts were extracted and categorized according to similar 
characteristics. Based on these categorized parts, a checklist was then developed to improve students’ EDs. In or- 
der to improve the validity of the checklist, three professionals and three graduate students in the area of science 
education revised the checklist by conducting several seminars. 

Methodology of the Second Stage 

Sample 

In the second stage, thirty-four 8" graders and thirty-one 9" graders in the science gifted center participated 


in this research voluntarily. They were divided into Groups A, B, and C randomly (Figure 1). To evaluate the effective- 
ness of the checklist, Groups A and B were experimental groups while Group C was included as the control group. 
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Group A (n = 25) 


8 students in grade 9 and 
17 students in grade 8 








Group B (n = 25) 


8 students in grade 9 and 
17 students in grade 8 








Group C (n= 15) 


15 students in grade 9 














L u 
Design an experimental procedure for the first task for 30 minutes 
{ J L 
Checklist with brief Noses 
Checklist explanation 


Compare and discuss their 


for 5 minutes first EDs 


Revise the first ED 
for 10 minutes Revise the first ED 


: Revise the first ED 
for 5 minutes 


for 10 minutes 


{ 1 { 


Design experimental procedure for the second task for 30 minutes 





























Figure 1: Overall process for evaluating the effectiveness of the checklist. 


Tasks and application of the checklist 


The overall process used to evaluate the effectiveness of the checklist is shown in Figure 1. In the first step of 
this process, students in all groups were asked to design an experimental procedure for the given task | (Figure 2) 
for 30 minutes without any help or guidance. For Task 1, the task utilized in Kim & Kim’s (2012) research to explore 
various characteristics of students’ EDs such as students’ perception of experimental goals, variables and experi- 
mental equipment chosen by students, and students’ difficulties in designing experimental procedures was used. 


[Context] The student was climbing in the cold winter and was left alone somewhere in mountains. The weather is very cold, 
the wind is strong and dry, and it takes a lot of time to be rescued. During that time, student must keep body warm. 


[Task] Design an experimental procedure to answer the following question: Would outerwear made of non-woven or vinyl better 
protect the body from the cold weather? 


Figure 2: Task 1. 


After the first EDs, Groups A and B were asked to revise their designs. At this time, only a checklist was pro- 
vided to students in group A without detailed description of the checklist. To students in Group B, a checklist was 
provided with a 5-minute brief description of the checklist. Groups A and B students modified their first EDs for 
5~10 minutes using the checklist. Group C was not given a checklist. Students were asked to compare and discuss 
their first EDs with peers and revise them for 10 minutes. 

After revision of their EDs, a new task 2 (Figure 3) was presented and students in all groups were asked to 
design anew experimental procedure without the checklist for 30 minutes. For Task 2 the task used in Park’s (2012) 
research to develop learning materials to improve students’ scientific creativity was used. 
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[Context] The figure shows a paper clip floating in the air by an electromagnet. There is a thin 
glass plate between the electromagnet and the paper clip. 


[Task] Design an experimental procedure to answer the following question: In this figure, 
what factors affect the strength of the electromagnet at point P? 





Figure3: Task 2. 


Data analysis 


There were 130 EDs by 65 students for two tasks. Student's ED ability was assessed using a checklist. Each item 
was assessed as 1 point when it was performed well, 0 point when it was performed normally, or -1 point when it 
was performed insufficiently. In order to secure the reliability of assessment results, results were compared using 
Cohen's kappa by one researcher and another science teacher who had a master’s degree in the area of science 
education. Cohen's kappa is a well-known method to test reliability by measuring agreement between two raters. 
A kappa value of +1 represents a perfect agreement between raters while a value of 0 represents that agreements 
are obtained by random chance (McHugh, 2012). High kappa value indicates high reliability. 

To evaluate the effectiveness of the checklist, t-test was first used to compare EDs for task 1 and task 2 for each 
group. While t-test was used for intra-group comparisons, ANCOVA was used for group-to-group comparisons. The 
reason for using ANCOVA was to eliminate differences caused by group participants to see influences only caused 
by checklist. For all statistical analysis, SPSS (21) was used. 


Results of the Main Research 
Results of the First Stage 
Students’ incomplete and inappropriate parts in EDs 


In the first step of the main research, there were 18 EDs written by 18 students without any help or guidance. 
From these designs, various incomplete and inappropriate parts were identified (Table 3). The number of cases 
of each difficulty existed was not recorded in this stage because results obtained from this stage were used to 
develop a checklist to improve students’ EDs. 


Table 3. Students’ incomplete and inappropriate parts found in their EDs. 


Code Incomplete and inappropriate parts with an example 


D1 Students set variables for the experimental goal or the hypothesis incorrectly. (E.g., after suggesting a hypothesis, ‘The 
more legs disperse the force, the stronger the bridge will be’, s/he sets variables irrelevant to the hypothesis as follows: ‘if 
the materials of the bridge are changed....’.) 


























D2 Students do not recognize the feasibility of the experiment. (E.g., a student designs the building to float off the ground to 
reduce the impact of the earthquake.) 

D3 Variables are hard to control. (E.g., it is difficult to keep the conditions, such as initial speed and angle, the same when 
students tried to fly paper planes under the same conditions.) 

D4 Some of the dependents or independent variables are omitted. 

D5 Concrete values of the variables are not specified concretely or completely. (E.g., when using ‘paper planes with the same 
shape but a different size of the wings’, there are no concrete values for the different sizes of the wings.) 

D6 Necessary operational definition of the variables is not defined. (E.g., when using the term, ‘dispersion of the forces’, the 
student does not define the term operationally, such as ‘the number of legs of the bridge’.) 

D7 Measurement methods are not specified concretely. 

D8 Necessary equipment is omitted. (E.g., when trying to fly a paper plane with different angles, 30°, 45°, 60°, ..., students do 


not specify the appropriate equipment to fly the paper plane.) 
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Code Incomplete and inappropriate parts with an example 
D9 Specifications of equipment are not given. (E.g., when using ‘acoustic insulation materials’, the students do not provide the 
concrete specification such as type or size.) 
D10 The experimental process is not clearly presented step by step. 
D11 Necessary figures or detailed explanations about them are missing. 


A checklist to improve students’ ED 


Based on results of students’ incomplete parts or mistakes in their EDs (Table 3), a checklist was developed to 
improve students’ ED as shown in Table 4. 


Table 4. A checklist to improve students’ EDs. 


Category Items to check 


1. Experimental goals 1-1. Does the experimental procedure meet the experimental goals or hypotheses? 





2-1. Is the experimental method possible in terms of spatial and temporal scale? 
2-2. Are the experiment preparations (materials or equipment) available? 

2-3. Is the value to be measured actually measurable? 

2-4. Is control of the control variables actually possible? 


2. Feasibility 





3-1. Have all the necessary variables been set? 

3-2. Are the values of variables specified? 

3-3. Are variables defined operationally, if necessary? 

3-4. Are there formulas, and calculation methods, if necessary? 


3. Variable setting 





4-1. Does it specify what should be measured? 
4-2. Does it specify the measurement methods clearly? 
4. Measurement 4-3. Is all required experimental equipment included? 
4-4. Are the specifications for the measurement equipment given, if necessary? 
4-5. Is there an iterative measurement process? 





5-1. Is the process described step-by-step to be followed? 
5. Concreteness 5-2. Is the picture clearly presented, if necessary? 
5-3. If the picture is presented, is it clearly explained? 


Results of the Second Stage 
Reliability of the checklist 


EDs from 65 students were scored by the researcher and another teacher using the checklist. As shown in 
Table 5 the Cohen's kappa value was 0.797 between the two evaluators without any consultation. After the first 
evaluation, results of these evaluations were compared with each other and difference between them was dis- 
cussed. After the discussion, scores were revised individually. The Cohen's kappa value of revised evaluations was 
0.866. For Cohen's kappa, agreement is considered ‘moderate’ when the kappa value is 0.60 ~ 0.79, ‘strong’ when 
the value is 0.80 ~ 0.90, and ‘almost perfect’ when the value is 0.90 or more (McHugh, 2012). Therefore, the reli- 
ability of the checklist was high. 
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Table 5. Cohen's kappa values for each item on the checklist. 














Kappa value 
Category Item 
First evaluation Second evaluation 
1 1-1 713 953 
2-1 622 809 
2-2 .793 839 
2 2-3 501 671 
2-4 647 749 
Average 641 167 
3-1 .953 953 
3-2 .939 954 
3 3-3 834 847 
3-4 853 853 
Average 895 902 
4-1 827 915 
4-2 .708 781 
4-3 .780 864 
: 4-4 .960 .960 
4-5 1.000* .950 
Average 855 894 
5 5-1 873 890 
5-2 791 919 
5-3 751 814 
Average 805 874 
Average 197 866 


Note: * Results of the two evaluators are completely identical. 


Results of the first evaluation showed that only one item (item 2-3) had a Cohen's kappa value of 0.60 or less. 
Eight items had values of 0.61 ~ 0.79 while seven items had values of 0.80 or more. By category, categories 1 and 
2 had values between 0.61 and 0.79 while all other categories had values over 0.80. Especially, given that item 
‘2-3’ and category 2 showed relatively low kappa values, it was not easy for evaluators to make a judgment on the 
feasibility of the experiment. For the second evaluation, two items had values of 0.61 ~ 0.79 while all other items 
had values of 0.80 or more. 

Therefore, it was confirmed that this checklist could be used as it is. However, it was more reliable when evalu- 
ators discussed their evaluation results for category 2. 


Effectiveness of the checklist 
In order to examine the effectiveness of the checklist, evaluation scores of the first ED for task 1 and the second 


ED for task 2 were compared using paired t-test. Results are shown in Table 6. Since the value of evaluation for each 
item ranged from -1 to +1, the mean value in Table 6 could have minus value when students’ EDs were ‘insufficient’ 
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The 1* exp. design 


SD 


Paired t-test between the first and the second EDs. 


The 2" exp. design 


M 


SD 


(P. 932-945) 


Effect 
size 


Group A 25 


Table 6. 
N 
Group B 25 
Group C 15 


4.47 
4.77 
4.40 


6.48 
9.44 
-1.27 


5.14 
4.37 


Paired t p 
7.53 <.01 
8.02 <.01 


1.71 
1.69 


0.04 > .05 


5.43 


0.01 


According to results shown in Table 6, group A students who received a checklist and group B students who 
received a checklist with brief explanation showed statistically significant improvement in scores of their second 
ED compared to their first ED. Effect sizes for groups A and B were also very high (1.71 and 1.69, respectively). 
Therefore, the checklist was very helpful in improving students’ EDs. However, for Group C students who did not 
receive the checklist, no improvement was seen in their second ED compared to their first ED. Results of more 
detailed analysis to find improvement for each item are shown in Table 7. 











Table 7. | Comparisons of scores between the first and the second EDs for each item of the checklist. 
Group A Group B 
Category Item Paired t Paired t 
1% ED 24 ED 1* ED 2-4 ED 

1 0.68 1.00 -232* 0.76 1.00 “1.81 
2-2 0.44 1.00 -3.22 0.64 0.84 -1.26 

2 2-3 0.60 0.48 52 0.76 0.84 -0.56 
2-4 -0.12 0.64 -3.93* 0.44 0.82 -2.00 
Average 0.40 0.78 -3.80** 0.65 0.88 -2.85* 
3-1 -0.84 0.04 -5.34** -0.08 0.80 -5.10** 
3-2 -0.88 0.04 -4.85** -0.92 0.28 -7.62** 

3 3-3 -0.28 -0.40 061 -0.84 -0.12 -3.22"* 
3-4 0.04 0.00 1.00 0.04 0.04 0.00 
Average -0.49 -0.08 -4.55** 0.45 0.25 -7.13* 
4-1 0.26 0.24 0.07 0.84 0.90 -0.46 
4-2 -0.40 0.08 -1.83 -0.16 0.34 -1.98 
4-3 -0.32 0.20 -2.03* -0.56 0.06 -2.59* 

: 4-4 -0.96 -0.28 -3.93"* -0.80 -0.32 -2.67* 
4-5 0.08 0.28 -1.87 0.08 0.58 -4.42** 
Average 0.27 0.10 -3.49** 0.12 0.31 410" 
5-1 0.68 0.78 -0.63 0.72 0.92 -1.37 
5-2 0.52 0.74 -1.06 0.04 1.00 -4.91* 

2 5-3 0.24 0.36 -0.60 -0.16 0.52 -3.92** 
Average 0.48 0.63 -1.30 0.20 0.81 -5.55** 


Note: Values represent mean values of the checklist score. 


*p <0.05, *p<0.01. 


As shown in Table 7, five items (2-3, 3-4, 4-1, 4-2, and 5-1) among seventeen items did not show any improve- 
ment in group A or group B. The followings might the reasons why it is difficult to improve these five items. First, 
regarding item 2-3, to judge the possibility of the experiment, conducting pre-experiment may be necessary for 
designing an experimental procedure before the main experiment. However, since the actual experiment was 
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not conducted in this research, students felt hard to predict whether measuring was possible. The need for pre- 
experiment was also seen for items 4-1 and 4-2. Students might not be able to determine values to be measured 
before an actual experiment (item 4-1). Likewise, the specificity of the measurement method (item 4-2) is also 
revised and supplemented in the course of actual experiment after ED. 

Second, the reason that there was no improvement in item 3-4 (necessary formulas or calculation methods) 
was probably because students did not have enough quantitative background knowledge related to the experi- 
ment. ED requires active interaction between methodological skills or techniques and background knowledge 
related to the experiment. Finally, the lack of step-by-step describing skills (item 5-1) might be related to students’ 
lack of ability to write. 

Therefore, it was found that pre-experiments, background knowledge, and writing skills were needed for 
more complete ED with checklists. 

Results of ANCOVA analysis comparing scores between groups by setting scores of the first ED as covariant 
variable are shown in Table 8. 


Table 8. Comparisons between groups using ANCOVA. 


Group Difference of M SD P 
Group A 
Group B -1.94 1.34 > .05 
Group C 7.68 1.51 <.01 
Group B 
Group A 1.94 1.34 >.05 
Group C 9.62 1.54 <.01 
Group C 
Group A -7.68 1.51 <.01 
Group B -9.62 1.54 <.01 


As shown in Table 8, there was no significant difference between groups A and B. However, there were statisti- 
cally significant differences between groups A and C or between groups B and C. Therefore, the checklist was very 
useful for improving students’ EDs. It was possible to help students design experiments by presenting a checklist 
only without providing any explanation. 


Discussion 


This research identified parts of ED that were difficult to students. It was found that a checklist developed 
based on these findings could actually improve students’ EDs. However, there are still many things to consider to 
use the checklist more effectively. 

Although the checklist alone was effective, additional elements were needed to help students design ex- 
periments. Even though ED was done before the actual experiment, pre-experiment was necessary to judge the 
possibility of the experiment, determine values of variables, and specify measurement methods in more details. 
‘Thinking (when designing experimental procedure)’ in scientific inquiry is closely related to actual ‘doing’ To un- 
derstand students’ processes or difficulties in designing experimental procedure and improve their EDs, we need to 
explore how students’ thinking in ED interacts with actual performing of experiments. Further research is needed 
to observe the broader process from pre-experiment to ED and main experiment. 

Additional elements also include mathematical formula and calculation method. In cases when direct mea- 
surements are impossible, it is often necessary to measure other variables and obtain desired values through 
calculations involving variables. For example, the height of a falling object can be measured to determine the 
speed of the object when it falls on the floor. In fact, from scientific observation to interpreting data and drawing 
conclusion, the scientific inquiry process is closely related to theories. In this research, it was also confirmed that 
design of experimental procedure was linked to theories. 

The last additional element to consider in ED is writing skill. In fact, writing is an important component of 
scientific inquiry. For example, ASE Science Teachers’ Handbook stresses that communication skills are key inquiry 
skills along with thinking skills and practical skills (Milner, 1986, p. 5). Many science educators have also emphasized 
the importance of writing skills in the scientific inquiry (e.g. Darian, 2003; Nam, Choi, & Hand, 2011; Wellington & 
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Osborne, 2001). However, Garcia-Mila, and Andersen (2007) have noted that writing has little effect in terms of 
improving scientific reasoning. Therefore, further studies are needed to determine the role of writing skills in ED 
improvement. 

To improve students’ EDs, we investigated difficulties that students had when designing an open inquiry. For 
this, only six tasks were used. However, EDs can change according to different tasks and types of inquiry. Therefore, 
we need to use more diverse types of tasks to identify students’ difficulties in designing experiments so that we can 
revise and develop a checklist based on new findings. To clarify various types of tasks and inquiry, consideration 
of authentic scientific inquiry is necessary. For instance, the process of scientific inquiry can be different between 
inductive process (to find out hidden features, draw a relationship between variables, or obtain certain specific 
values) and deductive process (to test a hypothesis or to confirm theoretical predictions). Many science educators 
(e.g., Alters, 1997; Lederman et al., 2014; Windschtl, 2004) have pointed that there is no universal procedure for 
scientific research. Cyclic procedures or non-linear procedures may appear in real scientific research (Park, Jang, 
& Kim, 2009). For example, after obtaining and analyzing data, variables to be measured or measuring methods 
can change. This process can also be repeated cyclically and some steps can be skipped. Therefore, according to 
different types of tasks and inquiry, different checklists may be necessary. 

In this research, no significant difference in ED was found between the group of students who were given 
only the checklist and the group of students who were given the checklist along with a brief explanation of the 
checklist. This means that the checklist alone is effective. It might also indicate that the explanation about the 
checklist might be too brief to have any effect. If more detailed explanations about the checklist with concrete 
examples for each item are given or if students can practice contents corresponding to each item in the checklist 
under an actual situation, it might be more effective for improving students’ EDs. In fact, we have already begun 
to develop and apply learning activities to help students understand the meaning of each item in the checklist 
and practice it in real situations. These new results will be reported in the near future. 


Conclusions 


For more authentic scientific inquiry, science educators have been emphasizing open inquiry that includes 
finding inquiry questions, designing experiments, conducting experiments, analyzing experimental results, and 
writing inquiry reports. However, there are many reports that students have difficulties in conducting open inqui- 
ries. From this point of view, this research might provide a practical way to improve students’ ability to conduct an 
open inquiry. In particular, this research was meaningful in that it analyzed difficulties that students encountered 
during inquiry process and found way to help them based on their difficulties. As a result, this research found that 
students could effectively design experiments using a checklist developed based on their difficulties. 

Furthermore, the checklist developed in this research can be used as a tool to examine student's ED process 
more closely. Although the present research only assessed whether students performed each sub-item of the ED 
well or not, it could be used as a framework to specifically explore how students would actually perform each item 
of the checklist. For example, for item 2-1 of the checklist, we can investigate what standard or criteria students 
would actually use to determine the possibility of the experimental method suggested by them. 

However, the checklist in this research is not perfect for improving students’ EDs. In other words, it should 
be supplemented by considering various types of inquiry, inquiry tasks, and student characteristics or levels. As 
mentioned in the discussion section, it is also necessary to study ways to improve students’ EDs by considering 
other factors such as pre-experiments, background knowledge, and writing skills. 

It is also worth mentioning that each step of the inquiry is closely related to each other. For example, inquiry 
design is related to inquiry problem while data interpretation is also related to ED. Therefore, in order to improve 
students’ actual performance of open inquiry, it is necessary to help them with the whole process, including vari- 
ous inquiry skills, from finding inquiry problems to writing an experimental report. 

Finally, it should be noted that the number of participants used in this research is not sufficient. To test the 
effectiveness of the checklist, the total number of participants was 65. However, the number of participants in 
each group was 15 ~ 25 which was not enough to generalize results of this research. Therefore, a future research 
involving more students is needed to confirm our results. Studies involving different grade levels and general 
students who are not gifted students are also be needed. In particular, this research was aimed at science gifted 
students. However, open inquiry is also emphasized for ordinary school students according to the National Sci- 
ence Curriculum in Korea. Therefore, difficulties in E for general students also needed to be determined to help 
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their ED abilities. Especially, difficulties that general students face in EDs might be more basic or more varied than 
gifted students. As a result, these follow-up studies are needed to make results of this research more generalizable. 
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